Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering (e.g., "What is Abraham Lincoln's birthday?"). However, do they answer such questions based on exposure to similar questions during training (i.e., cheating), or by genuinely learning to extract knowledge from sources like Wikipedia? In this paper, we investigate this issue using a controlled biography dataset. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data. $\textbf{Essentially}$, for knowledge to be reliably extracted, it must be sufficiently augmented (e.g., through paraphrasing, sentence shuffling) $\textit{during pretraining}$. Without such augmentation, knowledge may be memorized but not extractable, leading to 0% accuracy, regardless of subsequent instruction fine-tuning. To understand why this occurs, we employ (nearly) linear probing to demonstrate a strong connection between the observed correlation and how the model internally encodes knowledge -- whether it is linearly encoded in the hidden embeddings of entity names or distributed across other token embeddings in the training text. This paper provides $\textbf{several key recommendations for LLM pretraining in the industry}$: (1) rewrite the pretraining data -- using small, auxiliary models -- to provide knowledge augmentation, and (2) incorporate more instruction-finetuning data into the pretraining stage before it becomes too late.

This forecast is part of the Top 10 Stocks Package, as one of I Know First's systematic trading tools. Package Name: Stock Forecast & S&P500 Forecast Recommended Positions: Long Forecast Length: 3 Days (12/1/2020 – 12/4/2020) I Know First Average: 13.95% The algorithm correctly predicted 10 out 10 of the suggested trades in the Stock Forecast & S&P500 Forecast Package for this 3 Days forecast. The prediction with the highest return was KSS, at 27.02%. CCL and MT also performed well for this time horizon with returns of 17.37% and 17.18%, respectively.

An image released by Tesla Motors, is a conceptual design rendering of the Hyperloop passenger transport capsule. If it were anyone else, the notion of digging hundreds of miles of tunnels to create a new subterranean transportation network under congested cities would seem like pure science fiction. But the dreamer behind this vision is Elon Musk, the billionaire innovator who has already shown with his Tesla electric cars and SpaceX rockets that he thinks big and doesn't wait for others to transform fantasy into reality. Once again, Musk is aiming to shake up an arcane industry not used to outside-the-box thinking and yet potentially ripe for disruption: the underground world of tunneling. He could use tunneling to achieve his moonshot goal of clearing up Los Angeles traffic.